Cloud computing hosts large number of modern day applications using the virtualization concept. However, end-to-end network latencies are detrimental to the performance of IoT (Internet of Things) applications like video surveillance and health monitoring. Although edge/fog computing alleviates the latency concerns to some extent, it is still not suitable for applications having intercommunicating tasks. Further, these applications can be elastic in nature and demand more tasks during their life-time. To address this gap, in this paper a network aware co-allocation strategy for the tasks of an individual applications is proposed. After modelling the problem using bin packing approach with additional constraints, the authors propose a novel heuristic IcAPER,(Inter-communication Aware Placement for Elastic Requests) algorithm. The proposed algorithm uses the network neighborhood machine for placement, once current resource is fully utilized by the application. Using CloudsimPlus simulator the performance IcAPER algorithm is compared with First Come First Serve (FCFS), Random and First Fit Decreasing (FFD) algorithms for the parameters (a) resource utilization (b) resource fragmentation and (c) Number of requests having intercommunicating tasks placed on to same PM. Extensive simulation results shows IcAPER maps 34% more tasks on to the same PM and also increase the resource utilization by 13% while decreasing the resource fragmentation by 37.8% when compared to other algorithms in our consideration.