The Authentication Header Protocol is one of the original building blocks of IPsec, the security suite that protects IP traffic. It adds a verification tag to each packet so the receiver can confirm where the packet came from and whether anyone changed it in transit.
This article explains what the protocol does, how its header is structured, and how it differs from the encryption features that IPsec also offers. It also covers where the protocol fits in modern networks, why many deployments now prefer encrypted alternatives, and what practical checks an administrator should make before relying on it.
Why Packets Need a Way to Prove Their Origin
Every packet on an IP network carries a source address, but that address is just a number written into a header. Anyone with the right tools can write a different number there, a technique generally called address spoofing. A receiver that trusts the source field alone has no reliable way to tell a genuine packet from a forged one.
Authentication solves this by attaching proof that only the legitimate sender could have produced. The proof is a value computed from the packet contents and a secret shared between the two endpoints, so any change to the packet or use of the wrong secret produces a mismatch.
This matters most on networks where traffic crosses shared or untrusted links, such as the public internet or a wireless segment. Without such proof, an attacker placed between two hosts could alter a message or inject traffic that appears to come from a trusted peer.
What the Authentication Header Protocol Actually Does
The Authentication Header Protocol, usually abbreviated AH, is defined as part of the IPsec architecture. It inserts a dedicated header between the IP header and the payload. That header carries a sequence number, an identifier for the security association, and an integrity check value, which is a fixed-length code calculated with a keyed hash function.
The keyed hash is a mathematical function that takes the packet data and a secret key and returns a short fingerprint. Because the key is required to produce the fingerprint, a receiver that recomputes it and gets the same result can be confident the packet was built by a party holding that key and was not altered afterward.
The protocol provides data origin authentication, connectionless integrity, and optional protection against replayed packets through the sequence number. It does not hide the contents of the packet, so the data remains readable by anyone who can see the traffic.
Key Features and Where AH Leaves Gaps
The protocol is valued for a few specific properties, but it also has limits that matter in real deployments. Administrators should weigh both before choosing it for a given link or policy.
- Verifies that the packet was created by a holder of the shared key, supporting origin authentication.
- Detects modification of protected fields, so tampered packets are discarded by the receiver.
- Uses a sequence number that helps the receiver reject replayed copies of earlier packets.
- Does not encrypt the payload, so the content of protected traffic stays visible to observers.
- Can be disrupted by devices that rewrite addresses, since changed fields break the integrity check.
Because AH covers the packet’s IP header fields, network address translation, which rewrites source addresses at a gateway, typically causes verification to fail. This is one reason the protocol is rarely used alone on networks that depend on translation. Teams should test any planned path that includes address rewriting before committing to this design.
How AH Differs From the Encapsulating Security Payload
IPsec offers two main protocols for protecting packets. The Authentication Header Protocol provides integrity and authentication but no confidentiality. The Encapsulating Security Payload, usually called ESP, can provide encryption along with authentication and integrity in a single format.
This difference shapes how each protocol is chosen. A link that carries sensitive data needs confidentiality, which points toward ESP. A link where data may be public but must not be forged or altered can use AH, provided no address translation sits in the path.
In practice, ESP has become the more common choice because most organizations want both secrecy and verification. AH remains relevant in specific designs, and understanding it helps explain why IPsec separates these functions into distinct components.
Modes of Operation: Transport and Tunnel
The protocol can be applied in two modes. In transport mode, the header is inserted between the original IP header and the upper-layer payload, and the original addresses remain in place. This mode suits communication directly between two hosts that both support IPsec.
In tunnel mode, the entire original IP packet is treated as payload and wrapped inside a new IP header with new endpoint addresses. This is common when securing traffic between two gateways that protect networks behind them. Tunnel mode adds bytes to every packet, which can affect the maximum transmission unit and should be planned for.
Choosing a mode depends on where the security endpoints sit. Transport mode protects the host-to-host path, while tunnel mode protects a route between networks. Both modes rely on the same verification logic, so the mode decision mainly concerns addressing and overhead rather than the strength of the check.
Security Associations and Key Management
Before AH can protect traffic, the two peers must agree on the settings it will use. That agreement is called a security association, and it records the algorithm, the key, the mode, and the lifetime of the relationship. Each association is identified by a value carried in the header so the receiver knows which settings to apply.
Keys are typically established by a separate process such as the Internet Key Exchange, which negotiates shared secrets securely. Keeping keys fresh by rotating them on a schedule limits the damage if one is ever exposed, and most implementations allow lifetimes to be configured in time or traffic volume.
Administrators should avoid reusing weak or hard-coded keys. Using a strong, randomly generated secret of adequate length and storing it with restricted access keeps the verification meaningful. A keyed hash is only as trustworthy as the secrecy of its key.
Choosing Algorithms and Keeping Them Current
The integrity check depends on the hash algorithm chosen for the security association. The choice affects both strength and performance, and older algorithms can become unsuitable over time as cryptanalysis advances. Teams should select from currently recommended options and review those choices periodically.
Hash Algorithm Categories
Keyed hash functions based on the SHA-2 family are widely supported and are the usual recommendation for new deployments. Older options such as MD5 are considered weak for this purpose and should be avoided where the configuration allows a choice.
When comparing options, check the current guidance published by standards bodies and the documentation of your specific platform. Support varies between implementations, so a setting that works on one vendor’s device may not be offered on another.
Replay Protection and Sequence Numbers
The sequence number in the header lets a receiver detect duplicate packets. Each packet from a sender carries a number that increases, and the receiver maintains a window of acceptable numbers. Packets that fall outside the window or repeat an already-seen number are discarded.
Replay protection is valuable because an attacker who records legitimate traffic should not be able to resend it later and have it accepted. The window size and whether it is enabled are configurable, and turning the feature off removes this defense.
Performance and Hardware Considerations
Computing a keyed hash for every packet adds processing work. On modern hardware this cost is usually modest, but high-throughput links can reveal the difference. Some network interface cards and processors include acceleration for common hash functions, which can reduce the load.
Before deploying, measure throughput and latency on realistic traffic rather than relying on general estimates. Actual figures depend heavily on the device, the packet sizes, and the configuration, so results from a lab test are more useful than published claims.
Common Deployment Problems and Troubleshooting
The most frequent failure with this protocol is a verification error caused by a device that modifies the packet after it is protected. Address translation is the classic case, because it changes fields that the integrity check covers. The result is that packets are dropped at the receiver even though the keys are correct.
Mismatched settings are another common cause. If the two peers disagree on the algorithm, the mode, the key, or the security association identifier, the receiver cannot validate the packet. Comparing the configuration on both ends side by side often reveals the difference quickly.
Maximum transmission unit problems can also appear, especially in tunnel mode, where the added headers push packets past the link limit. Lowering the interface size or adjusting the maximum segment size for affected connections usually resolves symptoms such as stalled large transfers.
Security Limits You Should Understand
The protocol protects the integrity and origin of packets but does not make the traffic private. Anyone who can capture the packets can read their contents, including any sensitive information carried in the payload. Confidentiality must come from another layer, such as encryption provided by ESP or by the application itself.
The protocol also does not decide whether a peer should be trusted in the first place. Authentication proves that a packet came from a holder of a key, but the security of the whole arrangement depends on how that key was distributed and whether it stayed secret. A compromised key lets an attacker produce fully valid packets.
The protocol does not defend against denial-of-service attacks that flood a link with garbage. Verification can reject forged traffic, but the processing needed to reject it can still consume resources. Network-level filtering and rate controls remain necessary alongside any cryptographic protection.
Testing Whether Your Setup Works Correctly
Before relying on a configuration, confirm that protected traffic actually flows and that tampered traffic is rejected. A simple test is to establish the association between two lab devices, send ordinary traffic, and verify through the device logs and counters that packets are being authenticated.
A second test checks the failure path. Deliberately using a mismatched key or altering a setting should cause packets to be dropped, and the logs should show the reason. If a broken configuration still appears to work, the protection may not be applied to the traffic you intended to secure.
Repeat these tests after any change to firmware, routing, or address translation, since those changes can silently break verification. Keeping a short written test plan makes it easier to repeat the same checks each time.
Where the Protocol Fits in Modern Networks
Today the Authentication Header Protocol is mostly found in older or specialized deployments. Many organizations have moved to designs that combine encryption and authentication through ESP, or to transport-layer protections that operate above IP. Understanding AH is still useful for reading existing configurations and for evaluating legacy systems.
When a network uses IPsec, administrators should document which protocol protects each path and why. This record helps during audits and makes future migrations easier, since it shows exactly which links depend on verification-only protection.
Clear documentation also helps new staff understand why certain paths behave differently from others, and it prevents well-meaning changes that could quietly remove protection from traffic that still needs it.
Practical Checklist Before Using This Protocol
Use the following checks to decide whether this protocol fits a particular link and to avoid the most common problems before they reach production traffic.
- Confirm that the path contains no address translation or other devices that rewrite protected header fields.
- Verify that the traffic does not require confidentiality, since this protocol does not encrypt the payload.
- Select a current keyed hash algorithm and document the choice for both peers.
- Generate strong keys, store them securely, and set a rotation schedule that matches your risk level.
- Enable replay protection and test that duplicate packets are rejected as expected.
Once these points are settled, test the configuration on a non-production link and review the logs for both accepted and rejected packets. Revisit the decision whenever the network topology changes, because a path that worked before may now include a device that breaks verification.
Planning Your Next Steps With IPsec Protections
If you manage a network that uses IPsec, start by inventorying every security association and recording its protocol, mode, algorithm, and key lifetime. This inventory shows where verification-only protection is in use and highlights any paths that may need encryption added.
Next, review the devices along each protected route for address translation and maximum transmission unit limits, since these are the most frequent sources of failure. Plan the test cases described earlier and schedule them after each significant change.
Over the longer term, consider whether your links should move toward encrypted and authenticated designs that cover both needs in one configuration. Keeping your knowledge of these protocols current will help you make that transition with confidence.