Markov chain Monte Carlo algorithms are widely used in the fitting of generalized linear models (GLM). Such model fitting is somewhat of an art form requiring suitable trickery and tuning to obtain results one can have confidence in. A wide range of practical issues arise. The focus here is on parameter identifiability and posterior propriety. In particular, we clarify that non-identifiability arises for usual GLM's and discuss its implications for simulation based model fitting. Since often, some part of the prior specification is vague we consider whether the resulting posterior is proper, providing rather general and easy to check results for GLM's. We also show that if a Gibbs sampler is run with an improper posterior, it may be possible to use the output to obtain meaningful inference for certain model unknowns.