Assertive is not "aggressive" or "arrogant" and being unsportsmanlike is never acceptable. On the other hand, when not being assertive enough you can disrupt your own experience by relinquishing your line or space to anyone who comes near, either by pulling up, slowing down or moving far to the side. Others may be coming up behind you and being passive in your stroke makes you more vulnerable to sudden bumping from behind.
The best way to get used to being assertive in a crowd is to practice. Join a group swim squad or find a training buddy. Get used to swimming with others right beside you. Get used to bumping and getting knocked a bit. You'll gain confidence knowing you can handle it and discovering that the extra splashes aren't too bad after all.
Have your coach or buddy gently grab your ankle while you're swimming to get used to the sensation and how your assertive, consistent stroke and kick is all you need to shake them loose and keep on your pace. This can be done in a pool or open water with equal effect.
Position and Pace smartly. Create a strategy for your starting position and know your optimal pace. If you're racing to place or win and feel you need to be near the front, but still have concerns, start at the left or right edge of the front line, opposite from the first turn.
For example, if the first buoy means turning right, then start on the far left of the front line. That way you'll always be on the outside, with less traffic, and will avoid all the other swimmers squeezing down into your lane as everyone approaches the turn buoy. Being comfortable and able to race your own pace is worth being the extra 10 to 15 meters on the outside.
Remember: races aren't won in the swim and definitely not in the first 300 meters.
If you're not there to race for place, err on the side of caution. Start to the side or back, start slower and most importantly start at a comfortable pace. Otherwise you risk going out too hard and quickly, and losing your breath. Think of the first 200 to 500 meters as a warm-up if you need to and know that you can pick up the pace and get into rhythm after the initial frenzy is over. You'll be delighted by how many you'll pass later in the swim as they fade from starting too fast.
Fear: Unknown and Unfamiliar Environment
The cozy confines of your local pool, with its walls, ropes and lines at the bottom, make it easy to focus on what matters—your technique and pacing. But when suddenly faced with an unfamiliar lake or ocean with poor visibility, cold temperatures, rough water and supposed "lurking creatures" your focus is often drawn away from what matters and moves to creating drama and becoming unnerved by the unknown.
Whether irrational or real, you have a duty to yourself to learn how to remain steadfast and not relinquish any clout to the unknown and instead remain focused on what matters.