In short, EC2 security group (SG) is a set of ACCEPT firewall rules for incoming packets that can apply to TCP, UDP or ICMP. When an instance is launched with a given SG, firewall rules from this group are activated for this instance in EC2’s internal distributed firewall (it’s not the same as iptables on your instance!).
A common misconception is that SG rules can apply only to traffic from Internet into EC2 - this is incorrect, SGs apply to all traffic that is coming to your instance.
SG can be thought of as a security profile or a security role - it promotes good practice of managing firewall by role, not by machine. For example, you could say servers with “webapp” role must be able to connect to servers with “mysql” role on port 3306. Going further with security profile analogy, an instance can be launched with multiple SGs - similar to a server with multiple roles. Because all rules in SG are ACCEPT rules, it’s trivially easy to combine them (more on this down in my future features wishlist).
Each rule in SG (called “permission”) must specify the source of packets to be allowed. It can be either a subnet anywhere on the Internet (in CIDR notation, with 0.0.0.0/0 being entire Internet) or another security group, which once again promotes managing firewall by role. Interestingly, in the latter case, the source SG does not necessarily have to belong to your AWS account - it can be anyone’s. This makes it easy to grant selective access to your instances from instances run by your friends, partners and vendors. It works only if their instances are running in the same EC2 region (US or EU), because this functionality works only using EC2 private IP addresses.