The Griffiths phase has been proposed to induce a stretched critical regime that facilitates self organizing of brain networks for optimal function. This phase stems from the intrinsic structural heterogeneity of brain networks, such as the hierarchical modular structure. In this work, we extend this concept to modified hierarchical networks with small-world connections based on Hanoi networks. Through extensive simulations, we identify the role of an exponential distribution of the inter-moduli connectivity probability across hierarchies determining the emergence of the Griffiths phase. Numerical results and the complementary spectral analysis on the relevant networks can be helpful for a deeper understanding of the essential structural characteristics of finite dimensional networks to support the Griffiths phase.