Infrared small target detection (ISTD) is a heated topic in infrared image processing, and it is intensively applied in early warning systems and missile guidance. ISTD faces significant challenges such as low signal-to-noise ratio (SNR), small size, lack of distinct shape or structure, and weak texture, making it a demanding task. The performance of conventional object detection networks and semantic segmentation networks considerably deteriorates when applied directly to ISTD tasks. To address this issue, this paper proposes a new dual network collaboration-based image semantic segmentation network for ISTD, termed as DualNet. DualNet divides the task into two sub-tasks, namely reducing missed detections and reducing false alarms, with two sub-networks focusing on their respective targets (with cost reduced) by employing a weighted loss function to integrate sub-network information. DualNet effectively balances the miss detection rate and false alarm rate. Experimental results show that DualNet outperforms general neural network models (e.g. FCN, DeepLabv3, cGAN and U-net) on the ISTD task, with an improved F1-measure by 0.08. Furthermore, our model outperforms ACM and MDvsFA-cGAN, two most representative ISTD models based on deep learning, and several non-deep-learning-based ISTD methods.