DOS supported different video modes/resolution depending on the game and graphics card. See this section of the DOSBox wiki for some details on the video modes that DOSBox supports. For example, very early DOS games typically used CGA which was 4 colors and 320x200 resolution. Later, VGA was common -- 16 colors at 640x480 or 256 colors at 320x200. Other modes include EGA, which was between CGA and VGA, and SVGA (Super VGA) which came after VGA and supports up to 32-bit color at 1024x768 on DOSBox. Once you figure out what graphics mode(s) the game you want to run supports, you can change the setting in the dosbox.conf file to whatever it needs, if different than default. (Though the default SVGA is pretty backwards compatible, so you can often leave it on that.)

Pixelated graphics are to be expected to some extent with DOS games, but it will probably be worse if you're using a huge new LCD screen (especially if it's widescreen). Back in the DOS era 15" and 17" monitors were pretty common. I use a 17" 4:3 CRT for my fullscreen DOS gaming and it works fine, although you can still have fun with DOS games on an LCD monitor and get them looking pretty good if you don't need 100% historical accuracy. Getting certain games running on DOSBox can be tricky even after you've used it for a while. The trick to getting anything running properly in DOSBox almost always lies with the Dosbox.conf file, so familiarize yourself with that. You can have different .conf files for each game and use a command flag on the shortcut of the game so it knows to use that conf instead of the general dosbox.conf -- useful if you have several games that require something other than default settings.

If you don't know what needs to be changed for the game (especially if you don't have much experience with DOSBox yet), it's best to do some research on the internet and figure out what settings work best for that game -- Google "DOSBox" plus the name of the game plus "conf" or "cycles" or whatever you're trying to figure out. DOSBox has a pretty good Compatibility section on their website that will tell you how well a game is supported by the latest version of DOSBox and often has useful comments about the number of cycles that's best or other settings that may need to be changed.

Most common .conf settings that will need to be changed:
cycles (speed the game runs at, older games especially will need to be set manually to a lower number of cycles; newer games often work okay with auto -- this can be tricky to get just right and is often a matter of using the DOSBox hotkeys while running the game to change the number of cycles on the fly until the game 'feels right', which can be difficult if you've never played the game)
machine setting (graphics mode: CGA, EGA, VGA, SVGA, etc)
fullscreen (set to true if you want it to default to fullscreen. use ALT+Enter to toggle fullscreen and windowed mode at any time)

Since emulation is less efficient and requires more processing power than running directly on the native hardware, some later DOS games (mid-late 90s) can be a pain to get working at acceptable speeds on most machines, so don't feel bad if you can't get every game working flawlessly right away. Sometimes it's a choice of spending hours and hours on obscure forums trying to figure out which hidden settings you need to tweak to get the game playable or just giving up and doing something else. Thankfully most popular games will have plenty of information out there to help you configure DOSBox for them, so unless your tastes run very obscure, it shouldn't be too hard.