Some weren't scared so easily from what can be read.
Some were here to make a buck, build Georgian Architecture in Maryland or Virginia and many parts of states south, and send the majority of their gross income back to family nobility that financed their off shore endeavors.
Hence, all the estates their older brothers or parents built around Oxford and Cambridge, financed by crops, pig iron, or sugar via Slave Labour, or indentured servants abroad, and managed by cousins or siblings that were younger. People went back and forth often.
Try reading Ben Franklin's Autobiography as one example..