Trog is a common name for humans that have become infected with the Troglodyte Degeneration Contagion (or "TDC"), and slowly degenerated into scampering troglodytes who fear the light.
The Fallout 3 trogs reside in The Pitt (formerly Pittsburgh, Pennsylvania). Due to the town's infestation with the Troglodyte Degeneration Contagion, around 25% of the populace have transformed into these wild, chattering monstrosities, hiding in darkness and attacking anything on sight. However some of their idle chatter suggests that they still have some shred of intellect, such as murmuring thanks to the player when killed, or random words such as 'kill'. Still all evidence indicates their degeneration overpowers all their humanity, as they will kill even friends and loved ones with savage violence.
The following is based on Van Buren and has not been confirmed by canon sources.
The trogs of Van Buren differ greatly, especially in the sense that they retain a large portion of their intelligence. The trogs of Hoover Dam, while not pleasant to look at, look much more human than their Pitt counterparts. A trog's skin has a dull white color and their hair (in the case of those trogs who do have it) is a yellowish colour due to a deficiency of melanin. The trade-off is that their skin became thicker, making them more resilient to their hazardous environment. They are, however, very sensitive to light, a trait they share with their more animalistic brethren from The Pitt.
The Hoover Dam trogs were nomadic at one time thanks to a cave-in at their previous dwellings. Before Dodge and his troops came to Hoover, Billy-Bob and his trogs took up residence in Baseline, figuring it was pretty secure from caving in. No one argued with the trogs, and when Dodge came to Hoover, he asked if they would help out Otto with the maintenance in exchange for food, goods, and water. The trogs accepted and have been content ever since to mosey around Baseline making sure every light was flashing correctly.
As a direct result of their disease, trogs are hairless, naked, primate-like creatures with long, gangly arms and legs that crawl on all fours. Their skin appears raw and wrinkled, and their faces are scrunched into permanent scowls, with their teeth constantly bared. They have four fingers and four toes, and their feet appear to be in the process of turning into "hands", adding to their primate-like appearance. They have no external genitalia, or any visual clues to their gender, ruling out sexual reproduction.
Despite their savage appearance, trogs are capable of hunting in packs like wolves. While some may simply charge at the player head-on, others will attempt to sneak up on the player from behind or above. Their speed is almost on par with a yao guai, and they are able to leap even farther.
Trog packs usually circle their prey, slowly closing in, and the player can be quickly surrounded if they aren't careful. Although trogs are known to be extremely photosensitive, in-game trogs will actively pursue the player through light and dark alike. However, they will only spawn in darkened areas such as the Steelyard, Power Plant, and Abandoned Apartments; not in lit places such as Uptown even though they are one door away from its streets.
Trogs are no more durable than standard humans; a basic trog is on par with a basic feral ghoul, and even the powerful trog brutes and savages are still only on par with standard raiders in terms of health. However, they are much faster than normal humans, have fairly powerful melee attacks, and rarely hunt alone. Additionally, trogs are usually fought while the player is poorly equipped, as the player's equipment is taken away upon entering The Pitt, and good weapons and armor can be hard to come by in The Pitt until the player finishes the Arena and enters Uptown.
Regular trogs are the most common type of trog on normal difficulty. These are larger than trog fledglings, but are still vulnerable to headshots from high-powered weapons. In terms of damage and health they're on par with mole rats or basic feral ghouls.
Trog brutes are the most common trogs on very hard difficulty. They are only slightly larger than regular trogs, but have almost double the health. They will take more time and effort than usual to put down. The same rule applies: target the head. The legs can be crippled to slow their attacks to a crawl if a fight begins to get overwhelming. Trog brutes are actually on par with mid-level raiders and most standard human non-player characters in terms of health, although their melee attacks are fairly powerful.
These trogs are the smallest and weakest of the trog family, however they are still quite fast and may overwhelm an unaware player. Headshots from decent weapons are usually fatal. In fact they are one of the weakest creatures in the game (on par with bloatflies or radroaches in terms of health and attack damage). Players level 7 and below will primarily encounter this type of trog. They are easily dispatched with a .32 pistol and other small weapons.
These trogs are the strongest and most powerful trogs in The Pitt. They are encountered rarely, but are essentially trog brutes with slightly more health. It helps considerably to bring a high-powered gun such as Lincoln's repeater or the Blackhawk. Although only slightly tougher than trog brutes, trog savages have powerful melee attacks similar in damage potential to those of a giant radscorpion or mirelurk hunter.
It seems most trogs carry exactly one item despite wearing nothing to carry it in. Bobby pins are very common, as well as single doses of chems and various miscellaneous item.
When talking to Midea in her house for the first time, you can ask either what a trog is or imply that it is a fish. The player can also ask if they are like a super mutant. Which will result in Midea asking what a super mutant is.
When facing a trog, they will sometimes mutter words like "hate", "kill", and "food", suggesting that trogs have some semblance of their former humanity left, but are unable to fight their animalistic hunger. Some can even be heard muttering "...peace" or "thank you" right before they die.
When shooting a trog in the head, only the top half of their skull gets blown off as opposed to their whole head flying off or exploding, similar to the abomination, super mutant overlord and super mutant behemoth.
The word "troglodyte" is a Latin word meaning "cave-dweller" or "caveman", and is sometimes used to describe someone who is uneducated or uncivilized.
The concept of the trog is very different in The Pitt from the original concept in Van Buren. This can be attributed to being from different developers as well as there possibly being different forms of trogs. However since trogs from The Pitt are the only ones to appear in a published game they are the only ones that are canon.
Edit
It seems that headshots do not work on all types of trogs, doing the same amount of damage as a shot anywhere else on its body.
