Step 1: Modify the file: /usr/share/X11/xkb/symbols/us based on the fr file (in the same folder). The syntax is not hard to guess. I kinda copy the fr layout into it.
Step 2: Apply the change: sudo dpkg-reconfigure xkb-data. So the English keyboard will now got the French layout which matches my physical Keyboard, and ibus-unikey uses the English keyboard (welcome)
Step 3: Logout and Login again
Make sure to create a backup of /usr/share/X11/xkb/symbols/us before editing it.