We proposed a hp-HDG scheme for the Friedrichs' system where large classes of PDEs can be analyzed in this unified framework. The hp-HDG formulations for both one- and two-field structures are presented and are proved to be well-posed with few additional mild assumptions besides the conditions required by the Friedrichs' system. The key ingredient in dealing with non-conforming interfaces in HDG methods is to take advantage of nature built-in mortars. Moreover, the mortar technique can easily be realized by interpolation. In addition, we showed that with the proper choice of mortars and the approximation space of trace the scheme is conservative. Except for the analysis, several numerical experiments are also performed to validate the effectiveness of hp-adaptation. The testing cases cover elliptic, hyperbolic, and hybrid types of PDEs. Two approaches are implemented: One is a posterior type of error indicator while the other is an adjoint-based error estimator. These two approaches are found to be comparable. By either approach, we observed the improvement in convergence rates in some testing cases where high gradient, discontinuous, and/or singular features are involved in the solution. The numerically polluted area can be confined to a limited region as the adaption proceeds.