Fragmented ip packets always go to core 1

[HKalbasi]

In a ~70 Gbit/s traffic I have ~2 Gbit/s ip fragment packets, and all of them are going to the core 1 (and if I remove core 1 they will go to core 2), making core 1 have ~2 Gbit/s more traffic on average than other cores. I understand that since ports are not available in the ip fragment packets, we can't distribute them by 4 tuple. But is it possible to distribute them by 2 tuple instead of handing all of them to core 1? Alternatively is it possible to drop them by the NIC using a filter? I guess it shouldn't matter since retina can't defragment and use them anyway.