Although many chat-oriented dialogue systems are emerging and showing promising results, their performance is still limited and the dialogue with a system often breaks down. To counter dialogue breakdowns, I have been leading dialogue breakdown detection challenges, the aim of which is to develop algorithms to detect inappropriate utterances by the system so that dialogue breakdowns can be avoided. In this talk, I will go over the past challenges and mention the current performance of dialogue breakdown detection and the types of errors that still need to be tackled. In addition, I will describe the dialogue system live competitions that I have also been involved with, in which the audience, consisting mainly of researchers in the dialogue community, watch and evaluate a live dialogue conducted between users and dialogue systems. The motivation behind the event is to cultivate state-of-the-art techniques in chat-oriented dialogue systems and enable the dialogue community to share the problems with current dialogue systems. I will talk about the lessons learnt from organizing the event and mention possible directions for the dialogue systems community.